Description

As of today MySQL server which is controlled by VTTablet, starts in read-only mode. This is because Vitess controls the MySQL cluster topology and chooses one tablet to be the primary for each shard, and only that one MySQL server should be read-write. All other servers in that shard are replicas, and should not be taking any writes. However this doesn't prevent users like root and vt_dba who have SUPER privileges from writing to the database on replicas and causing errant GTIDs. We want to leverage GLOBAL super_read_only configuration in order to protect against these scenarios. This will make sure that apart from primary no other component or offline system can mutate DB resulting in errant GTIDs that are then lying in wait to cause later failures, as you can see in #9312 and #10094

Furthermore, it is possible that due to program bugs or unexpected failures during PRS/ERS, we end up with a replica that is not in read-only mode. With super-read-only change we will make sure that we cover those cases as well.

Tablet schema refactoring

This change is built on top of #11520, where instead of using withddl we use a declarative approach to initializing vttablet's sidecar schema. Using a declarative approach helps us condense all our schema changes to one place, which allows us to make changes to the code to apply super-read-only to the DB.
The previous approach of using withddl meant that schema changes were scattered all across the code, which made it extremely difficult to do the super-read-only change.

Details

This PR usese #11520 as a building block. I am listing major changes below to help you review the PR.

• mysql57.cnf & mysql80.cnf contains super-read-only, to ensure all Mysql instances comes up in super-read-only mode.
• During init_db.sql we switch super-read-only OFF temporarily in order to perform some mutations like creating necessary users and permissions.
• Since VTTestServer doesn't need super-read-only, therefore we need to come up with a separate SQL initialization (init_testserver_db.sql) for it.
• Adding # {{custom_sql}} inside init_db.sql file. This is because in our tests we modify this file and add custom SQL like (change password of DBA user etc). Now that we expect setting super-read-only ON to execute at the end, we need these custom sqls to be added before that last line. See example: go/test/endtoend/backup/vtbackup/main_test.go
• During PRS & ERS, we make sure that only primary turns off super-read-only and rest of replica are strictly in super-read-only mode.
• Adding 'SuperReadOnly' property, which is available through VtctldClient GetFullStatus.

For Reference

Some related work items done in the past

#11706
#10094
#9312
#10448

Unresolved Issues during code changes:

I have filed few issues which I will need to resolve once I check-in this PR.

• Remove read_only usage from code base #12143

Related Issue(s)

Fixes #10363
Fixes #12180
Fixes #12140

Test Results

vtctld command to show super_read_only state

 vtctldclient --server localhost:15999 GetFullStatus zone1-0000000101
{
  "server_id": 493124066,
  "server_uuid": "ed6d02e6-a35b-11ed-b467-9cf2e7b0ba93",
  "replication_status": null,
  "primary_status": {
    "position": "MySQL56/ed6d02e6-a35b-11ed-b467-9cf2e7b0ba93:1-26",
    "file_position": "FilePos/vt-0000000101-bin.000001:14540"
  },
  "gtid_purged": "MySQL56/",
  "version": "8.0.31",
  "version_comment": "Homebrew",
  "read_only": false,
  "gtid_mode": "ON",
  "binlog_format": "ROW",
  "binlog_row_image": "FULL",
  "log_bin_enabled": true,
  "log_replica_updates": true,
  "semi_sync_primary_enabled": true,
  "semi_sync_replica_enabled": true,
  "semi_sync_primary_status": true,
  "semi_sync_replica_status": false,
  "semi_sync_primary_clients": 1,
  "semi_sync_primary_timeout": "1000000000000000000",
  "semi_sync_wait_for_replica_count": 1,
  "super_read_only": false
}

Checklist

• "Backport to:" labels have been added if this change should be back-ported
• Tests were added or are not required
• Documentation was added or is not required

Deployment Notes